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Data Assimilation of Real Data 



Time 



Observing System Simulation Experiment 


Applications of OSSEs 


1. Estimate effects of proposed instruments (and their competing 
designs)on analysis skill by exploiting simulated environment. 


2. Evaluate present and proposed techniques for data assimilation 
by exploiting known truth. 


Requirements for an OSSE 


1 . A simulation of “truth”, generally produced as a free-running 
forecast produced by an NWP model (termed the “Nature Run”). 

2. A simulation of a complete set of observations, drawn from truth. 

3. A simulation of observational instrument plus representativeness 
errors to be added to the results of step 2. 

4. A data assimilation system to ingest the results of step 3. 

All these steps must be done well enough to produce believable and 
useful metrics from step 4. 


Choice of a Nature Run 


1 . A good simulation of nature in all important aspects 

2. Ideally, individual realizations of the NR should be 
indistinguishable from corresponding realizations of 
nature (e.g., analyses) at the same time of year. 

3. Since a state-of-the-art OSSE will require a cycling DAS, 
the NR should have temporal consistency. 

4. For either 4DVAR or FGAT 3D VAR, or for high spatial resolution, 
NR datasets should have high frequency (i.e. ,<6 hours) 

5. Since dynamic balance is an important aspect of the 
atmosphere affecting a DAS, the NR datasets should 
have realistic balances. 

6. For these and other reasons, using a state-of-the-art NWP 
model having a demonstrated good climatology to produce 
NR data sets is arguably the best choice. 


Simulations of Nature (Nature Runs) 


NR Source 

ECMWF 

GMAO 

Availability 

now 

Summer 2014 

Horizontal Resolution 

35 km 

7 km 

# Vertical levels 

91 

72 

Full Period 

1 year 

2 year 

Output Frequency 

3 hourly 

0.5 hourly 

Other Characteristics 

03 

16 aerosols, CO, C02, 03 

Data set size 

2 TB 

2500 TB 


Simulation of Observations 


1 . Any observation that can be assimilated can be simulated! 

| y = H z { z)| 

2. Differences between the H used to assimilate and simulate 
will be interpreted by the DAS as representativeness errors 

t R = H(x f [z]) -H z { z) 

3. Therefore, as more realism is modeled for the observation 
simulation compared to the assimilation, more 
representativeness error is introduced, including gross errors 
that must be identified by the DAS quality control. 

4. It may be necessary to compensate for deficiencies in the nature 
run (e.g., unrealistic cloud cover) when simulating observations. 


Simulation of Observation Errors 


1 . When simulating observations, there is no instrument and 
thus no instrument error. It therefore needs to be explicitly 
simulated and added, unless it is negligible. 

2. An error of representativeness is generally implicitly created: 

e R = H(x t [ z]) - H z ( z) 

3. The real representativeness error is likely underestimated by 

the implicit component. Additional representativeness error needs 
to be simulated. 

4. If real observation errors are correlated, the simulated ones 
must also be if the OSSE is to verify. 

5. Errors effectively removed by the DAS may not require careful 
simulation! 


Probabilities of radiances being affected by clouds 


Effective radiative surface is a high-level cloud 


Ph= f P(H\f h )P(f h ) df h 

JF h 


Effective radiative surface is a medium-level cloud 


Pm = [ f P([M\H]\fm ) [1 - P(H\f h )] P(fmJh) dfm dfh 

J F h J F m 


Effective radiative surface is a low-level cloud 


Pi, = i [ f P([L\H n M]\fi) [1 - P(H\f h ) - P(M\f m , //,)] P(fi J m , fh) df, df m df h 

JF h JF m JF, 


P(H | f h ) 



Channel 


AIRS 



Locations of QC-accepted observations for AIRS channel 295 at 18 UTC 12 July 


Simulated 



Real 






Simulation of observations 


RAOB observation lunch time and final pressure determined by 
corresponding real observation 

RAOB trajectory determined from NR wind fields 

RAOB significant levels determined from NR sounding 

SATWINDS determined from locations with NR clouds or 
moisture gradients 


All relevant metrics depend on system errors 


Analysis error 

A = Ai B.B.R.R. Q) 


Background or Forecast error error 

B = B( A, Q, M) 


Observation error 


R = R(E, F) 


Daley and Menard 1 993 MWR 


Observational Error Simulation 


All observations have added random error with tuned variances 
Portions of added errors for: 

RAOB soundings are vertically correlated 
AMSUA, MHS are horizontally correlated 
SAT WINDS vertically and horizontally correlated 
AIRS and IASI channel correlated 
No mean errors added 


No gross errors added 


Assimilation System 


GEOS-5 (GMAO) DAS/Model 
NCEP/GMAO GSI (3DVAR) scheme 
Resolution: 55km, 72 levels 

Evaluation for 1-31 July 2005, with 2 week, accelerated spin up in June. 

Observations include all “conventional” observations available in 2011 
(except for precipitation rates) and all radiance available from AMSU-A, 
MHS, HIRS-4, AIRS, IASI instruments, plus GPSRO. 


Validation of OSSEs 


As for any simulation, OSSE results apply to the assimilation 
of real data only to the degree the OSSE for such an application 
validates well with regard to relevant metrics. 

OSSE validity is first determined by carefully comparing a 
variety of statistics that can be computed in both the real 
assimilation and OSSE contexts. 

Since data assimilation is a fundamentally statistical problem, 
OSSE validation and application must generally be statistical. 
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Standard deviations of QC-accepted y-H(xb) values (Real vs. OSSE) 
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Horizontal correlations of y-H(xb) 
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Correlations Between Channels of AIRS Innovations 
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Square roots of zonal means of temporal variances of analysis increments 
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Anomaly Correlation 


OSSE vs Real Data: Forecasts 


NH Anomaly Correlation: July SH Anomaly Correlation: July 



U-Wind RMS error: July 


U, 30N-90N U, 30S-90S U, 30S-30N 



Solid lines: 24 hour RMS error vs analysis 
Dashed lines: 120 hr forecast RMS error vs analysis 


July Adjoint: dry error energy norm 


Adjoint Observation Impact 
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Warnings 

Past problems with some OSSEs 


1 . Some OSSEs use a very reduced observation set of as a control 

2. Some OSSEs have very limited validation 

3. Some OSSEs are based on very limited “case studies” 

4. Some OSSEs use unrealistic observation errors (e.g., no rep. error) 

5. Some OSSEs use a deficient NR 


Warnings 

General criticisms of OSSEs 


1 . In OSSEs, the NR and DAS models are generally too alike, 
therefore underestimating model error and yielding 
overly-optimistic results. 

2. When future specific components of the observing systems 
are deployed, the system in general will be different as will 
the DAS techniques, and therefore the specific OSSE results 
will not apply. 


3. OSSEs are just bad science! 


Response to Warnings 


1. Design OSSEs thoughtfully. 

2. Consider implications of all assumptions. 

3. Validate OSSEs carefully. 

4. Beware of quick shortcuts. 

5. Specify reasonable observation error statistics. 

6. Avoid conflicts of interest. 

7. Avoid over-selling results. 

8. Only attempt to answer appropriate questions. 

9. Be skeptical of others’ works. 

10. Be critical of your own work (This is science!). 


Fractional reduction of zonal means of temporal variances of analysis errors 

compared with background errors 
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From validation experiment reported last year 



Fractional reduction of zonal means of temporal variances of analysis errors 

compared with background errors 
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Square roots of zonal means of temporal variances of analysis increments 
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Characteristics of Real Observation Errors 


1 . Generally unknown 

2. Even statistics not well known 

3. Often biased 

4. Correlated (maybe even with background error) 

5. Include gross errors 

6. Generally non-Gaussian 

(a result of 5 or some basic physics; e.g. nonlinearity) 
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